ISA 444: Business Forecasting

.title[
# ISA 444: Business Forecasting
]
.subtitle[
## 08: Forecasting Environment
]
.author[
### <br>Fadel M. Megahed, PhD <br><br> Professor <br> Farmer School of Business<br> Miami University<br><br> <a href="https://twitter.com/FadelMegahed"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z" /></svg> <span class="citation">@FadelMegahed</span></a> <br> <a href="https://github.com/fmegahed/"><svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z" /></svg> fmegahed</a> <br> <a href="mailto:fmegahed@miamioh.edu"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z" /></svg> fmegahed@miamioh.edu</a><br> <a href="https://calendly.com/fmegahed"><svg viewBox="0 0 384 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M202.021 0C122.202 0 70.503 32.703 29.914 91.026c-7.363 10.58-5.093 25.086 5.178 32.874l43.138 32.709c10.373 7.865 25.132 6.026 33.253-4.148 25.049-31.381 43.63-49.449 82.757-49.449 30.764 0 68.816 19.799 68.816 49.631 0 22.552-18.617 34.134-48.993 51.164-35.423 19.86-82.299 44.576-82.299 106.405V320c0 13.255 10.745 24 24 24h72.471c13.255 0 24-10.745 24-24v-5.773c0-42.86 125.268-44.645 125.268-160.627C377.504 66.256 286.902 0 202.021 0zM192 373.459c-38.196 0-69.271 31.075-69.271 69.271 0 38.195 31.075 69.27 69.271 69.27s69.271-31.075 69.271-69.271-31.075-69.27-69.271-69.27z" /></svg> Automated Scheduler for Office Hours</a><br><br>
]
.date[
### Spring 2025
]

---

## Quick Refresher of Last Class

✅ Explain the differences between wide vs. long format

✅ Use [seaborn](https://seaborn.pydata.org/generated/seaborn.relplot.html) to plot multiple time-series

✅ Convert a data set to Nixtla's long format (`unique_id`, `ds`, `y`)

✅ Use [UtilsForecast](https://nixtlaverse.nixtla.io/utilsforecast/index.html) to visualize multiple series

---

## Learning Objectives for Today's Class

- Install and import Nixtla's libraries ([StatsForecast](https://nixtlaverse.nixtla.io/statsforecast/index.html), [MLForecast](https://nixtlaverse.nixtla.io/mlforecast/index.html), [NeuralForecast](https://nixtlaverse.nixtla.io/neuralforecast/docs/getting-started/introduction.html), [UtilsForecast](https://nixtlaverse.nixtla.io/utilsforecast/index.html), and [TimeGPT](https://nixtlaverse.nixtla.io/nixtla/docs/getting-started/introduction.html)) for forecasting

- Distinguish fixed window from rolling-origin

- Introduce forecast accuracy metrics (MAE, MAPE, RMSE)

---
class: inverse, center, middle

# The Nixtlaverse Open-Source Forecasting Libraries

---

## Nixtla's Forecasting Libraries

---

## Nixtla's Forecasting Libraries

Nixtla provides **several open-source Python libraries** (and a closed source **TimeGPT** tool accessible via API calls) for **scalable forecasting tasks**. These libraries are **relatively** easy to use and can be integrated into your future forecasting workflows:

- **[StatsForecast](https://nixtlaverse.nixtla.io/statsforecast/index.html)** – Fast & scalable statistical models (`ARIMA`, `ETS`, etc.).
- **[MLForecast](https://nixtlaverse.nixtla.io/mlforecast/index.html)** – Machine learning-based forecasting (e.g., `XGBoost`, `LightGBM`).
- **[NeuralForecast](https://nixtlaverse.nixtla.io/neuralforecast/docs/getting-started/introduction.html)** – Deep learning models for time series (e.g., `NBEATS`, `NHITS`, and `TFT`).
- **[UtilsForecast](https://nixtlaverse.nixtla.io/utilsforecast/index.html)** – Utility functions for plotting, evaluation, etc.
- **[TimeGPT](https://nixtlaverse.nixtla.io/nixtla/docs/getting-started/introduction.html)** – an AI transformer-powered forecasting API that requires minimal tuning.

These libraries enable **forecasting at scale, which you will need in practice**.

**Note:** These libraries can be installed via `pip` and are described in detail in [nixtlaverse.nixtla.io](https://nixtlaverse.nixtla.io/). Use the left-hand navigation bar to explore the documentation for each library.
]

---
class: inverse, center, middle

# Fixed Window vs. Rolling-Origin

---

## The Fixed Window Evaluation Approach

- **Fixed Window** is the simplest approach to splitting your time series data into a training and a testing/holdout set.

- The **goal** is to **train your model on the training set** and **evaluate its performance on the testing set (the last `k` observations in the data)**.  
  + .black[.bold[Note that this is quite different than traditional machine learning applications for cross sectional data.]]

- The evaluation on the **testing/holdout** set can serve two purposes:  
  + **Model Evaluation**: Assess the model's **performance on unseen data** (since it is not used during training, and hence acts as a proxy for the model's performance on future data).  
  + **Model Selection**: Compare the **performance of different models** to select the **best one**.   
     - Note that using this approach for model selection is **reasonable if your models do not involve hyperparameter tuning** (otherwise, you may overfit to the testing set).

---

## The Fixed Window Evaluation Approach

---

## The Rolling-Origin Evaluation Approach

- **Rolling-Origin Evaluation** is a method for splitting time series data into training and testing sets where the testing sets move forward over time.

- The **goal** is to **train the model on a subset of past observations** and evaluate its **performance on a future testing set at multiple time steps**.

- The key difference from a fixed window approach is that the **testing set shifts forward, allowing for multiple evaluations**:  
  + The **training set may expand (expanding window)** or **remain fixed (rolling window)**.  
  + This ensures that model performance is assessed across **different points in time**.  
  + It **reduces sensitivity to the initial split point** and provides a more **robust evaluation** of model performance over time.

---

## Expanding Window Evaluation (By 1 Month)

---

## Expanding Window Evaluation (By 12 Month)

---

## Rolling Non-Expanding Window Evaluation

---

## Cross Validation within the Nixtlaverse

**Note:** The `cross_validation` method can also be applied to other NixtlaForecast objects (e.g., `MLForecast`, `NeuralForecast`) to perform cross-validation for machine learning and deep learning models. See the [StatsForecast Cross Validation Tutorial](https://nixtlaverse.nixtla.io/statsforecast/docs/tutorials/crossvalidation.html) to access the page shown above.
]

---

## Activity: The `cross_validation` Method

---

## Recap of Fixed vs. Rolling-Origin

- **Fixed Window**:  
  + **Simplest approach** to splitting data into training and testing sets.  
  + **Testing set is fixed** and **does not move forward** over time.  
  + **Provides a single evaluation** of model performance.
  
- **Rolling-Origin**:
  + **Testing set moves forward** over time.  
  + **Training set may expand or remain fixed**.  
  + **Provides multiple evaluations** of model performance.  
  + **Reduces sensitivity to the initial split point**.  
  + **More robust evaluation** of model performance over time.

---

## Recap of Fixed vs. Rolling-Origin

- In practice, the **rolling-origin approach is preferred** for time series forecasting tasks since it mimics the real-world scenario of forecasting future data points. The choice of:  
  + **Expanding vs. Rolling Window**,
  + **Window Size**, and
  + **Step Size**
depends on the specific forecasting task and the data at hand.

---

# Forecast Accuracy Metrics

---

## Model Performance Evaluation in the Nixtlaverse

``` python
from utilsforecast.losses import *
```

### `evaluate`

``` python
evaluate (df:~AnyDFType, metrics:List[Callable],
           models:Optional[List[str]]=None,
           train_df:Optional[~AnyDFType]=None,
           level:Optional[List[int]]=None, id_col:str='unique_id',
           time_col:str='ds', target_col:str='y',
           agg_fn:Optional[str]=None)
```

**Source:** [Nixtla's UtilsForecast Evaluation Documentation](https://nixtlaverse.nixtla.io/utilsforecast/evaluation.html)
]

---

## Model Performance Evaluation in the Nixtlaverse (Cont.)

.font80[
<table><thead><tr><th></th><th><strong>Type</strong></th><th><strong>Default</strong></th><th><strong>Details</strong></th></tr></thead><tbody><tr><td>df</td><td>AnyDFType</td><td></td><td>Forecasts to evaluate.<br/>Must have <code>id_col</code>, <code>time_col</code>, <code>target_col</code> and models’ predictions.</td></tr><tr><td>metrics</td><td>List</td><td></td><td>Functions with arguments <code>df</code>, <code>models</code>, <code>id_col</code>, <code>target_col</code> and optionally <code>train_df</code>.</td></tr><tr><td>models</td><td>Optional</td><td>None</td><td>Names of the models to evaluate.<br/>If <code>None</code> will use every column in the dataframe after removing id, time and target.</td></tr><tr><td>train_df</td><td>Optional</td><td>None</td><td>Training set. Used to evaluate metrics such as <a href="https://Nixtla.github.io/utilsforecast/losses.html#mase" target="_blank" rel="noreferrer"><code>mase</code></a>.</td></tr><tr><td>level</td><td>Optional</td><td>None</td><td>Prediction interval levels. Used to compute losses that rely on quantiles.</td></tr><tr><td>id_col</td><td>str</td><td>unique_id</td><td>Column that identifies each serie.</td></tr><tr><td>time_col</td><td>str</td><td>ds</td><td>Column that identifies each timestep, its values can be timestamps or integers.</td></tr><tr><td>target_col</td><td>str</td><td>y</td><td>Column that contains the target.</td></tr><tr><td>agg_fn</td><td>Optional</td><td>None</td><td>Statistic to compute on the scores by id to reduce them to a single number.</td></tr><tr><td><strong>Returns</strong></td><td><strong>AnyDFType</strong></td><td></td><td><strong>Metrics with one row per (id, metric) combination and one column per model.<br/>If <code>agg_fn</code> is not <code>None</code>, there is only one row per metric.</strong></td></tr></tbody></table>
]

**Source:** [Nixtla's UtilsForecast Evaluation Documentation](https://nixtlaverse.nixtla.io/utilsforecast/evaluation.html)
]

---

## Losses

The most important train signal is the forecast error, which is the difference between the observed value `$y_{\tau}$` and the prediction `$\hat{y}_{\tau}$`, at time `$y_{\tau}$`:

`$$e_{\tau} = y_{\tau} - \hat{y}_{\tau} \quad \quad \tau \in \{t+1, \dots, t+H\}$$`

The train loss summarizes the forecast errors in different evaluation metrics.

**Source:** [Nixtla's UtilsForecast Losses Documentation](https://nixtlaverse.nixtla.io/utilsforecast/losses.html)
]

---

## Scale-Dependent Errors: `mae`

**MAE** measures the relative prediction accuracy by averaging the absolute deviations between forecasts and actual values.

`$$\text{MAE}(y_{\tau}, \hat{y}_{\tau}) = \frac{1}{H} \sum_{\tau = t+1}^{t+H} |y_{\tau} - \hat{y}_{\tau}|$$`
- **Interpretation**: Provides a straightforward measure of forecast accuracy; lower MAE indicates better performance.

- **Characteristic:** Does not penalize larger errors more than smaller ones; treats all errors equally.

**Source:** [Nixtla's UtilsForecast Losses Documentation](https://nixtlaverse.nixtla.io/utilsforecast/losses.html)
]

---

## Scale-Dependent Errors: `rmse`

**RMSE** is the square root of the average of the squared differences between forecasts and actual values.

`$$\text{RMSE}(y_{\tau}, \hat{y}_{\tau}) = \sqrt{\frac{1}{H} \sum_{\tau = t+1}^{t+H} (y_{\tau} - \hat{y}_{\tau})^2}$$`

- **Interpretation**: Emphasizes larger errors due to squaring; useful when large errors are particularly undesirable.

- **Characteristic:**  Penalizes large errors more than MAE (i.e., more sensitive to ourliers compared to MAE).

**Source:** [Nixtla's UtilsForecast Losses Documentation](https://nixtlaverse.nixtla.io/utilsforecast/losses.html)
]

---

## Percentage Errors: `mape`

**MAPE** calculates the average absolute error as a percentage of actual values.

`$$\text{MAPE}(y_{\tau}, \hat{y}_{\tau}) = \frac{1}{H} \sum_{\tau = t+1}^{t+H} \left| \frac{y_{\tau} - \hat{y}_{\tau}}{y_{\tau}} \right|$$`
**Interpretation**: Expresses forecast accuracy as a percentage; lower MAPE indicates better performance.

**Characteristic:** Can be misleading if actual values are close to zero, leading to extremely high MAPE values.

**Source:** [Nixtla's UtilsForecast Losses Documentation](https://nixtlaverse.nixtla.io/utilsforecast/losses.html)
]

---

# Recap

---

## Summary of Main Points

By now, you should be able to do the following:

- Distinguish fixed window from rolling-origin

- Introduce forecast accuracy metrics (MAE, MAPE, RMSE)

---

## 📝 Review and Clarification 📝

1. **Class Notes**: Take some time to revisit your class notes for key insights and concepts.
2. **Zoom Recording**: The recording of today's class will be made available on Canvas approximately 3-4 hours after the session ends.
3. **Questions**: Please don't hesitate to ask for clarification on any topics discussed in class. It's crucial not to let questions accumulate.